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ABSTRACT 



Acadia University (Nova Scotia) has completed year two of 
the Acadia Advantage initiative, which involved 1,500 students and 170 
faculty using IBM notebook computers in the curriculum. By year 2000, all 
4,000 students at Acadia University will have notebook computers and will be 
taking part in technology- enhanced courses. To assist faculty, the university 
developed a modular Automated Courseware Management Environment (ACME) which 
allows an instructor to easily create online courseware. The Online Testing 
Module (OTM) is one of the most successful modules, allowing faculty to 
prepare World Wide Web-based tests that include multiple choice, short 
answer, and essay questions, incorporating formatted text, images, video, and 
audio. The module is template-based, and users create tests that can be 
randomized, graded, and results posted both to the students and instructor. 
The package has proved so successful that more than 1,000 online tests are 
taken and marked daily. Students using the testing package for tutorial 
assessment have responded very favorably to the ease of access and the 
ability to monitor their own progress. This paper describes access control 
and security, test taking, creating and editing a test, question types, 
formatting a test, test management and statistics, and programming. Five 
figures present sample screens. (Author/DLS) 
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Abstract: Acadia University has completed year two of the Acadia Advantage initiative 
which involved 1500 students and 170 Acadia faculty using IBM notebook computers in 
the curriculum. By year 2000 all 4000 students at Acadia University will have notebook 
computers and will be taking part in technology-enhanced courses. To assist faculty, 
Acadia University developed a modular Automated Courseware Management 
Environment (ACME) which allows an instructor to easily create online courseware. The 
Online Testing Module (OTM) is one of the most successful modules, allowing faculty to 
prepare web-based tests that include multiple choice, short answer, and essay questions 
that incorporate formatted text, images, video, and audio. The module is template based 
and users create tests that can be randomized, graded, and results posted both to the 
students and instructor. Little or no computer experience is required to create a test. At 
Acadia University the package has proven so successful that we routinely have 1000 plus 
online tests taken and marked daily. Students using the testing package for tutorial 
assessment have responded very favorably to the ease of access, and the ability to 
monitor their own progress. 



Introduction 

There are many programs available for creating computer based testing, many of which allow users to create 
tests that can be placed on the web. The complexity of the tests is usually proportional to the complexity of the 
program used to create the test; simple multiple choice questions are easy to create using any number of 
programs. More complex tests using various forms of multimedia (images, sounds, and video), and short answer 
or essay questions, usually require expensive and difficult to master software packages. Adding the 
functionality of randomization, access control, and the ability to edit on-the-fly can add enough complexity to a 
package that it often requires the efforts of Instructional Designers to create the test or to work closely with the 
instructor requiring the test. 

The AITT (Acadia Institute for Teaching and Technology) undertook the development of ACME (Automated 
Courseware Management Environment) in the summer of 1996. At the onset of development, instructors at 
Acadia University were clear about the requirements of a web-based testing and tutorial component. Allowing 
students to access databanks of tests for self-evaluation, and allowing instructors to view the results of these 
self-evaluation tests was critical. The Online Testing Module (OTM) in ACME solved this problem for the 
instructors by allowing them to easily create online tests that satisfied their immediate requirements. Every 
component of the functionality of ACME is available through any Internet connection, allowing instructors to 
prepare their courses from anywhere in the world. 
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Online Testing Module 



OTM was designed as an enhancement to regular classroom courses, and not as a complete computer based 
training solution. At the onset of development the hardware and software of all users of the system was known, 
however a web browser is the only base requirement by both the instructor and student depending upon the type 
of multimedia used in the tests. Ongoing development is occurring due to constant feedback from the instructors 
using the system. 

In the past, an instructor's first indication that a student was struggling in their course may have occurred only 
after the first midterm. Weekly tutorials and tests in OTM are providing instructors with the information they 
require to determine how well students are grasping the course information, and to allow early intervention 
when required. 

In order to make use of web-based tests and tutorials instructors were insistent that OTM must allow them to 
create tests that went far beyond simple multiple choice questions, allowed the use of multimedia, were easy to 
create, and could provide instant feedback to the students. The online testing module, has been working for 4 
semesters, and currently has hundreds of tests in the system. On October 15th, 1997, the Online Testing Module 
of ACME delivered, and marked 1309 individual tests in a 24 hour period. 



Access Control and Security 

Access to the ACME system is restricted to instructors and registered students by the use of usernames and 
passwords. Each individual with access to ACME has a profile on the system that indicates the courses they can 
edit as an instructor, or those that they can participate in if they are a student or guest. The administrator creates 
instructor profiles that link the individual with their course, and provide editing rights to that course. Student 
profiles are generated automatically from the university registrars' database. Guest profiles can also be created 
to allow individuals outside of the university to participate in a course. The instructor can also restrict access to 
individual test folders within a course. 



Taking a Test 

A student wishing to take a test simply authenticates into the ACME system, selects a course from their profile 
and goes into the OTM module by clicking the "TEST" button at the bottom of the course page (see figure 1). 




Figure 1: A typical course page in ACME. 
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From a student's perspective the OTM module has only one choice; select and take a test. The test presented to 
the student can be a combination of Short Answer, Multiple Choice and Long Answer questions (figure 2). 
After completing and submitting a test, students may be provided with instant feedback (if the instructor has 
enabled that option), with their grade and the answers to each question. The program grades multiple choice, 
true or false, and short answer questions. Long answer or essay questions are stored in a special marking folder 
to be graded later by the instructor. 
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1) [5] 



What is this? 




2) [5] 



What do the four elements most abundant in life (carbon, oxygen, 
hydrogen and nitrogen) have in common? 

C a) They all have the same number of valence electrons. 

O b) Each element exists in only one isotopic form. 

C c) They are equal in electronegativity. 

O d) They are elements produced only by Hving cells. 

O e) They are all relatively light elements and can exist singly or in 
combination with other atoms as a oas. 
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4) [5] 



What is the process seen in the video 



t 




O traffic 

O cytoplasmic streaming 



Figure 2: A student view of a test in OTM showing questions that 
include images, formatted text, audio and video. 



Creating and Editing a Test 

Instructors must first create a folder, then put a test in that folder. Any number of folders and tests can be 
created. After creating a test, content must be added. The user-interface for creating and editing tests is shown 
in figure 3. 

After creating a test, the basic test information can be changed. The name of the test, a time limit for the test, 
whether the answers are provided to the students when they complete the test, and whether the test can be 
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retaken are all in the control of the instructor. As a student could take a test from any location, passwords can 
be assigned to those tests that the instructor wants to give in-class. The student must then be in class to hear the 
password. 



Edit Test Information 
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Figure 3: user-interface for creating and editing tests. 



A current items window (figure 3) shows the number and type of questions in a test. Each type of question is 
shown by an abbreviation; MC, SA and LA representing Multiple Choice, Short Answer and Long Answer 
respectively. This window is empty if no questions have been added to the test. Individual questions can be 
edited or removed after selecting them in the Current Item window. 



The number and type of questions added to the test determines the layout of a test. Section or title headings can 
be added anywhere throughout the test, and questions can be put inside random blocks. A random block allows 
a small number of questions to be randomly selected from a larger test bank when a test is chosen. Any number 
of random blocks can be added to a test, allowing for maximum test customization. 

A full preview of all the questions in a test is available to the instructor, with the option to edit a question 
directly from that full preview. A student preview is provided also which differs from the full preview in that 
there is no option to edit each question. Also if the questions are randomized, the randomization will appear in 
the student preview. 



Short answer. Multiple Choice and Long Answer Questions 

Instructors create questions by completing a template (figure 4) for each question type, and can add a question 
anywhere within a pre-existing test. 



Add Short Answer Question 

WeijjfefcF 

Qmprtjon Trxt; Ogqrtjrdfcy | 



"3 

..j 



Pti s cftjl# Arisvna r* : 



Matdttbjg; 




Figure 4: Short Answer question template. 
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The value of the question can be selected and the text of the question typed into the "Question Text” window. 
Short answer questions allow up to four possible answers. There is also an option to allow the answers to be 
case sensitive or case insensitive, or to have a variation in spelling. The selections of the possible answers and 
the case sensitivity determine how the ACME system will grade the question. Multiple choice and long answer 
questions are created in a similar manner, with the option of providing one or more correct answers in the 
multiple choice, and a comment to the student depending upon the answer they choose. 



Formatting a Test 

To add formatted text, images or other forms of multimedia such as sound and video to a test, an instructor 
clicks on an "Open Editor" button to open the AcmeScript editor. This semi-WYSIWYG editor was created to 
aid instructors in the formatting of their tests. The editor was written in JavaScript to be executed by the client. 
All forms of media, other than plain or formatted text, are located on a different web server to the one on which 
OTM resides. Links are made to these other media formats using the editor. Any media or file-type that can be 
supported by a browser plug-in will appear in-line in the test, other files will launch a helper application. 



Test Management and Statistics 

Instructors can view test statistics that are generated each time a student takes a test. The statistics include the 
number of times each student has taken a test, which students in a class have not taken a test (useful for large 
classes), and also statistics on each question. For multiple choice questions the statistics include the number of 
times the question was taken and how many times a student selected each answer. Short answer questions also 
include the number of times the question was answered, and the top five answers provided by students. This 
allows the instructor to quickly determine problem areas that the students may be having, or highlight poorly 
worded questions. 

Tests are managed using the test management window (figure 5). The order of the tests and test folders can be 
changed by selecting the "ORDER" button, selecting a test by clicking on the radio button beside the test name, 
and then choosing to move the test up, down, to the top or bottom. Tests created in one course can easily be 
copied or exported to another course, at which time test information can be edited, and questions added or 
deleted. 
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Figure 5: OTM test management window. 



Programming 



Combining all of the features of OTM with a transaction-based World Wide Web (WWW) interface and an 
operating system -independent storage and execution mechanism has proven to be a less than trivial task. 

The languages used for development of OTM, were PERL 5, HTML 3.2, and JavaScript 1.1. As PERL is 
implemented on the vast majority of operating systems, it makes a very suitable environment for implementing 
the required programs and libraries on the server side of the OTM system. Currently, ACME and OTM are 
running on Solaris 2.5.1, Solaris 2.6, SCO OpenServer 5, and Linux 2.x. We are also now working on a 
Windows NT, and a Windows 95 port. 

JavaScript was chosen as the language for writing additional client-side functionality because it is the native 
language for programming Netscape Navigator. Netscape is currently the most popular WWW browser in use, 
and is available for the majority of operating systems, thus increasing the platform independence of the 
client. To provide even further client independence, we are making an effort to ensure all our client code is 
compatible with the ECMAScript specification. 

To further increase portability, all files are stored as plain text. Initially attempts to use database formats such 
as dbm/gdbm or SQL databases were made, but each format was problematic. The dbm databases are not 
entirely independent of the operating system. When an operating system upgrade is performed, or when ACME 
data is moved from one operating system to another, dbm style databases tend to break, due to different versions 
of dbm libraries used. SQL databases were rejected because many organizations wishing to use ACME may not 
have somebody trained in the administration and use of such databases. However, all file access in OTM is 
done using a small file access library to allow it to use a database with minimal effort, and will eventually be an 
option in the configuration of ACME. 

The tests themselves are stored using a rather simple mechanism. Each course has its own directory in OTM. 
Inside each course are directories representing each test folder created by the instructor. Inside each folder are 
files containing the information for each test and the results of students writing each test. 

The automatic test generation and the automated marking are the most complex functions of OTM. The 
advanced testing features permitted by OTM, such as randomization, fuzzy matching, passwords, and time 
limits, add to this complexity. To handle these problems efficiently, lookup tables and small separate files 
rather than one large file have been used to reduce search times, and the processing times of critical sections. 
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